Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap

Monitoring machine learning models once they are deployed is challenging. It even more challenging to decide when retrain in real-case scenarios labeled data beyond reach, and monitoring performance metrics becomes unfeasible. In this work, we use non-parametric bootstrapped uncertainty estimates SHAP values provide explainable estimation as a technique that aims monitor the deterioration of deployment environments, well determine source model deteri- oration target labels not available. Classical methods purely aimed at detecting distribution shift, which can lead false positives sense has deterio- rated despite shift distribution. To estimate construct prediction intervals using novel bootstrap method, improves previous state-of-the-art work. We show both our detection system method achieve better than current state-of-the-art. Finally, AI techniques gain an understanding drivers deterioration. release open Python package, doubt, implements pro- posed methods, code used reproduce experiments.